Ensuring occupant awareness in vehicles

ABSTRACT

In general, techniques are described for ensuring occupant awareness in vehicles. A device comprising a processor may be configured to perform the techniques. The processor may be configured to determine an event which will change an operating characteristic of the vehicle, and determine an extent of time remaining until an estimated occurrence of the expected event. The processor may also be configured to determine a process by which to inform the occupant of an operating context of the vehicle based on the determined extent of time remaining until the estimated occurrence the expected event, and perform the process by which to inform the occupant of the operating context of the vehicle. The device may also include a memory configured to store the process.

TECHNICAL FIELD

This disclosure relates to vehicles and, particularly, to restoringoccupant awareness in vehicles.

BACKGROUND

Vehicles are increasingly becoming more autonomous. That is, vehiclesare beginning to perform tasks that an occupant would normally performwithout any occupant interaction. Levels of autonomy for vehicles havebeen defined with level zero generally indicating no automation up tolevel four or five, which may refer to a fully autonomous vehicle wherean individual need only specify a destination to which the fullyautonomous vehicle is to drive.

Currently, most production vehicles fall between levels zero and five.Mid-level (e.g., levels two through three) autonomous vehicles mayperform some tasks normally performed by an occupant when operating thevehicle using adaptive cruise control, providing lane monitoring, andperforming automated crash avoidance (usually by applying the brakes),etc.

In mid-level autonomous vehicles and even in fully autonomous (e.g.,level four or five) vehicles, when problems arise for which the vehicleis not equipped to handle, the autonomous vehicle may transition controlof the vehicle back to the occupant. The occupant may then operate thevehicle until the problem has been overcome.

SUMMARY

In general, the disclosure describes techniques for restoring occupantawareness in autonomous vehicles such that the occupant can resumeoperation of the vehicle, e.g., when problems arise for which thevehicle is not equipped to handle.

In one example, the disclosure describes a method of providinginformation to ensure occupant awareness in a vehicle, the methodcomprising determining, by one or more processors, an expected eventwhich will change an operating characteristic of the vehicle, anddetermining, by one or more processors, an extent of time remaininguntil an estimated occurrence of the expected event. The method alsocomprises selecting, by the one or more processors, a process of aplurality of processes by which inform the occupant of an operatingcontext of the vehicle based on the determined extent of time remaininguntil the estimated occurrence of the expected event, and performing, bythe one or more processors, the process by which to inform the occupantof the operating context of the vehicle.

In one example, the disclosure describes a device configured to provideinformation to ensure occupant awareness in a vehicle, the devicecomprising one or more processors configured to determine an expectedevent which will change an operating characteristic of the vehicle, anddetermine an extent of time remaining until an estimated occurrence ofthe expected event. The one or more processors are further configured todetermine a process by which to inform the occupant of an operatingcontext of the vehicle based on the determined extent of time remaininguntil the estimated occurrence the expected event, and perform theprocess by which to inform the occupant of the operating context of thevehicle. The device may also include a memory configured to store theprocess.

In one example, the disclosure describes a device configured to provideinformation to ensure occupant awareness in a vehicle, the devicecomprising means for determining an expected event which will change anoperating characteristic of the vehicle, and means for determining anextent of time remaining until an estimated occurrence of the expectedevent. The device further comprises means for determining a process bywhich to inform the occupant of an operating context of the vehiclebased on the determined extent of time remaining until the estimatedoccurrence of the expected event, and means for performing the processby which to inform the occupant of the operating context of the vehicle.

In one example, the disclosure describes a non-transitorycomputer-readable storage medium having stored thereon instructionsthat, when executed, cause one or more processors to determine anexpected event which will change an operating characteristic of thevehicle, determine an extent of time remaining until an estimatedoccurrence of the expected event, determine a process by which to informthe occupant of an operating context of the vehicle based on thedetermined extent of time remaining until the estimated occurrence ofthe expected event, and perform the process by which to inform theoccupant of the operating context of the vehicle.

The details of one or more examples are set forth in the accompanyingdrawings and the description below. Other features, objects, andadvantages will be apparent from the description, drawings, and claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an example autonomous vehicleconfigured to perform various aspects of the occupant awarenesstechniques described in this disclosure.

FIG. 2 is a block diagram illustrating various aspects of the vehicle ofFIG. 1 in more detail.

FIG. 3 is a diagram illustrating example operation by the processselection unit shown in FIG. 2 in selecting a process by which to ensureoccupant awareness of the operating context of the vehicle of FIG. 1 inaccordance with various aspects of the techniques described in thisdisclosure.

FIG. 4 is a diagram illustrating an example of a head-mounted displaysystem configured to perform various aspects of the techniques describedin this disclosure.

FIGS. 5A and 5B are diagrams showing example forward views havingdifferent levels of emphasis applied in accordance with various aspectsof the techniques described in this disclosure.

FIG. 6 is a flowchart illustrating example operation of the vehicle ofFIG. 1 in performing various aspects of the techniques described in thisdisclosure.

FIG. 7 is another flowchart illustrating example operation of thevehicle of FIG. 1 in performing various aspects of the techniquesdescribed in this disclosure.

DETAILED DESCRIPTION

As an occupant of an autonomous vehicle becomes less involved in theoperation of the vehicle, the occupant may become distracted and/orcomplacent, spending no more time on driving tasks than is necessary.However, autonomous vehicles may transfer control of the vehicle to theoccupant. For example, an autonomous vehicle may transfer control of thevehicle to the occupant when problems occur for which the autonomousvehicle is not equipped to handle. The occupant assuming control may bereferred to as a “primary occupant” as this occupant is primarilyresponsible for assuming control, e.g., when the autonomous vehicle mayno longer safely autonomously control the vehicle.

Transitioning control of the autonomous vehicle to the primary occupantmay occur suddenly, such as when equipment used to provide theautomation fails due to weather, malfunction and the like. Suddentransitions of control to a potentially distracted primary occupant maynot give the primary occupant time to regain sufficient context in whichto safely operate the vehicle.

Various aspect of the techniques set forth in this disclosure mayfacilitate restoring occupant awareness in autonomous vehicles when thevehicle has automated some aspect of the driving. The techniques mayensure (or, in some instances, restore) occupant awareness by providinginformation to the occupant so that the primary occupant and potentiallyother occupants may monitor the current driving context should theprimary occupant be suddenly (e.g., within minutes or, in someinstances, seconds) put in control of the vehicle. Such driving contextmay allow the primary occupant to more safely operate the vehicle uponreceiving sudden control.

In some examples, various aspects of the techniques may tailorrestoration of occupant awareness based on various criteria. Forexample, the techniques may restore occupant awareness based on time totransition (or, in other words, handoff of) control of the autonomousvehicle, the current state of the occupant, or a condition triggeringthe handoff event or any combination of the time to handoff, the stateof the occupant, and the type of condition trigging the handoff event.

FIG. 1 is a block diagram illustrating components of an exampleautonomous vehicle 10 configured to perform various aspects of theoccupant awareness techniques described in this disclosure. In theexample of FIG. 1, autonomous vehicle 10 may represent a vehicleconfigured to automate one or more tasks associated with operation ofvehicle 10, including automating most if not all of the tasks associatedwith operation of vehicle 10 such that an occupant need not, under mostconditions, maintain awareness of a context in which vehicle 10 isoperating.

Autonomous vehicle 10 is assumed in the description below to be anautomobile. However, the techniques described in this disclosure mayapply to any type of vehicle capable of conveying one or more occupantsand being autonomously operated, such as a motorcycle, a bus, arecreational vehicle (RV), a semi-trailer truck, a tractor or other typeof farm equipment, a train, a plane, a helicopter, a drone, a personaltransport vehicle, and the like.

In the example of FIG. 1, autonomous vehicle 10 includes a processor 12,a graphics processing unit (GPU) 14, and system memory 16. In someexamples, processor 12, and GPU 14 may be formed as an integratedcircuit (IC). For example, the IC may be considered as a processing chipwithin a chip package, and may be a system-on-chip (SoC).

Examples of processor 12, and GPU 14 may include fixed functionprocessing circuitry and/or programmable processing circuitry, and mayinclude, but not be limited to, one or more digital signal processors(DSPs), general purpose microprocessors, application specific integratedcircuits (ASICs), field programmable logic arrays (FPGAs), or otherequivalent integrated or discrete logic circuitry. Processor 12 may bethe central processing unit (CPU) of autonomous vehicle 10. In someexamples, GPU 14 may be specialized hardware that includes integratedand/or discrete logic circuitry that provides GPU 14 with massiveparallel processing capabilities suitable for graphics processing. Insome instances, GPU 14 may also include general purpose processingcapabilities, and may be referred to as a general purpose GPU (GPGPU)when implementing general purpose processing tasks (i.e., non-graphicsrelated tasks).

Processor 12 may execute various types of applications. Examples of theapplications include navigation applications, vehicle controlapplications, scheduling application, safety applications, web browsers,e-mail applications, spreadsheets, video games, or other applicationsthat generate viewable objects for display. System memory 16 may storeinstructions for execution of the one or more applications. Theexecution of an application on processor 12 causes processor 12 toproduce graphics data for image content that is to be displayed.Processor 12 may transmit graphics data of the image content to GPU 14for further processing based on instructions or commands that processor12 transmits to GPU 14.

Processor 12 may communicate with GPU 14 in accordance with a particularapplication processing interface (API). Examples of such APIs includethe DirectX® API by Microsoft®, the OpenGL® or OpenGL ES® by the Khronosgroup, and the OpenCL™; however, aspects of this disclosure are notlimited to the DirectX, the OpenGL, or the OpenCL APIs, and may beextended to other types of APIs. Moreover, the techniques described inthis disclosure are not required to function in accordance with an API,and processor 12 and GPU 14 may utilize any technique for communication.

System memory 16 may be the memory for vehicle 10. System memory 16 maycomprise one or more computer-readable storage media. Examples of systemmemory 16 include, but are not limited to, a random access memory (RAM),an electrically erasable programmable read-only memory (EEPROM), flashmemory, or other medium that can be used to carry or store desiredprogram code in the form of instructions and/or data structures and thatcan be accessed by a computer or a processor.

In some aspects, system memory 16 may include instructions that causeprocessor 12 to perform various aspects of the functions ascribed inthis disclosure to processor 12. Accordingly, system memory 16 may be acomputer-readable storage medium having instructions stored thereonthat, when executed, cause one or more processors (e.g., processor 12)to perform various functions.

System memory 16 is a non-transitory storage medium. The term“non-transitory” indicates that the storage medium is not embodied in acarrier wave or a propagated signal. However, the term “non-transitory”should not be interpreted to mean that system memory 16 is non-movableor that its contents are static. As one example, system memory 16 may beremoved from autonomous vehicle 10, and moved to another device. Asanother example, memory, substantially similar to system memory 16, maybe inserted into autonomous vehicle 10. In certain examples, anon-transitory storage medium may store data that can, over time, change(e.g., in RAM).

As further shown in the example of FIG. 1, autonomous vehicle 10 mayinclude a display 20 and a user interface 22. Display 20 may representany type of passive reflective screen on which images can be projected,or an active reflective, emissive, or transmissive display capable ofprojecting images (such as a light emitting diode (LED) display, anorganic LED (OLED) display, liquid crystal display (LCD), or any othertype of active display). Although shown as including a single display20, autonomous vehicle 10 may include a plurality of displays that maybe positioned throughout the cabin of autonomous vehicle 10. In someexamples, passive versions of display 20 or certain types of activeversions of display 20 (e.g., OLED displays) may be integrated intoseats, tables, roof liners, flooring, windows (or, in vehicles with nowindows or few windows, walls) or other aspects of the cabin ofautonomous vehicles. When display 20 represents a passive display,display 20 may also include a projector or other image projection devicecapable of projecting or otherwise recreating an image on passivedisplay 20.

Display 20 may also represent displays in wired or wirelesscommunication with autonomous vehicle 10. Display 20 may, for example,represent a computing device, such as a laptop computer, a heads-updisplay, a head-mounted display, an augmented reality computing deviceor display (such as “smart glasses”), a virtual reality computing deviceor display, a mobile phone (including a so-called “smart phone”), atablet computer, a gaming system, or another type of computing devicecapable of acting as an extension of, or in place of, a displayintegrated into autonomous vehicle 10.

User interface 22 may represent any type of physical or virtualinterface with which a user may interface to control variousfunctionalities of autonomous vehicle 10. User interface 22 may includephysical buttons, knobs, sliders or other physical control implements.User interface 22 may also include a virtual interface whereby anoccupant of autonomous vehicle 10 interacts with virtual buttons, knobs,sliders or other virtual interface elements via, as one example, atouch-sensitive screen, or via a touchless interface. The occupant mayinterface with user interface 22 to control one or more of a climatewithin autonomous vehicle 10, audio playback by autonomous vehicle 10,video playback by autonomous vehicle 10, transmissions (such ascellphone calls, video conferencing calls, and/or web conferencingcalls) through autonomous vehicle 10, or any other operation capable ofbeing performed by autonomous vehicle 10.

User interface 22 may also represent interfaces extended to display 20when acting as an extension of, or in place of, a display integratedinto autonomous vehicle 10. That is, user interface 22 may includevirtual interfaces presented via the above noted HUD, augmented realitycomputing device, virtual reality computing device or display, tabletcomputer, or any other of the different types of extended displayslisted above.

In the context of autonomous vehicle 10, user interface 22 may furtherrepresent physical elements used for manually or semi-manuallycontrolling autonomous vehicle 10. For example, user interface 22 mayinclude one or more steering wheels for controlling a direction oftravel of autonomous vehicle 10, one or more pedals for controlling arate of travel of autonomous vehicle 10, one or more hand brakes, etc.

Autonomous vehicle 10 may further include an autonomous control system24, which represents a system configured to autonomously operate one ormore aspects of vehicle 10 without requiring intervention by an occupantof autonomous vehicle 10. Autonomous control system 24 may includevarious sensors and units, such as a global positioning system (GPS)unit, one or more accelerometer units, one or more gyroscope units, oneor more compass units, one or more radar units, one or more LiDaR (whichrefers to a Light Detection and Ranging) units, one or more cameras, oneor more sensors for measuring various aspects of vehicle 10 (such as asteering wheel torque sensor, steering wheel grip sensor, one or morepedal sensors, tire sensors, tire pressure sensors), and any other typeof sensor or unit that may assist in autonomous operation of vehicle 10.

In this respect, autonomous control system 24 may control operation ofvehicle 10, allowing the occupant to participate in tasks unrelated tothe operation of vehicle 10. As the occupant of autonomous vehicle 10becomes less involved in the operation of vehicle 10, the occupant maybecome distracted and/or complacent, spending no more time on drivingtasks than is necessary. However, autonomous vehicle 10 may transfercontrol of vehicle 10 to the occupant when problems occur for whichautonomous vehicle 10 is unequipped to handle. The occupant assumingcontrol may be referred to as a “primary occupant” as this occupant isprimarily responsible for assuming control when autonomous vehicle 10may no longer safely autonomously control vehicle 10.

Transitioning control of autonomous vehicle 10 to the primary occupantmay occur suddenly, such as when equipment used by autonomous controlsystem 24 fails due to weather, malfunction and the like. Suddentransitions of control to a potentially distracted primary occupant maynot give the primary occupant time to regain sufficient context in whichto safely operate vehicle 10.

Transferring control may refer to returning responsibility of control tothe occupant such that the occupant directs operation of vehicle 10(e.g., using a steering wheel, pedals and other implements intraditional vehicles, or using various non-traditional interfaces, suchas virtual interfaces that allow for acceleration, braking, andsteering). Transferring control may also be only a partial transfer ofcontrol depending on the circumstances. For example, a sensor allowingfor determination of a rate of travel may fail, but all other autonomouscontrol systems for steering and navigation may be operational. In thisexample, vehicle 10 may transfer control of acceleration and braking butretain control of steering and navigation. As such, vehicle 10 maytransfer control of the operation of vehicle 10 when responsibility forone or more autonomously performed operations are transferred to theoccupant.

In accordance with various aspects of the techniques described in thisdisclosure, autonomous vehicle 10 may ensure occupant awareness by, asone example, providing information so that the primary occupant andpotentially other occupants may monitor a current driving context shouldthe primary occupant be suddenly put in control of vehicle 10. Suchdriving context may allow the primary occupant to more safely operatethe vehicle upon a transfer of control from autonomous control system 24to the primary occupant.

In operation, processor 12 may determine an event for which the primaryoccupant of the vehicle is to be apprised. In the context of autonomousvehicle 10 capable of controlling operation of vehicle 10, processor 12may determine, when autonomous control system 24 is currentlyautonomously controlling operation of vehicle 10, that autonomouscontrol system 24 is unable to continue to autonomously control theoperation of vehicle 10. When autonomous control system 24 determinesthat autonomous control of vehicle 10 is no longer feasible (e.g., givena failure of one or more sensors, occurrence of inclement weather, orsome other factor that may prevent successful operation of vehicle 10 byautonomous control system 24), autonomous control system 24 may indicateto processor 12 that an expected event is imminent, where such expectedevent may include some event that will change an operatingcharacteristic (such as whether autonomous control system 24 or theoccupant is responsible for control of vehicle 10). The expected eventmay, as one example, include transfer of control of the operation ofvehicle 10 to the primary occupant. The transfer of control event mayalso be referred to as a “handoff event,” where transfer of control fromautonomous control system 24 to the primary occupant may be referred toas “handoff.”

Autonomous control system 24 may, prior to signaling the handoff event,determine an extent of time remaining until the expected handoff event.Autonomous control system 24 may determine the extent of time based onany number of factors, including one or more of a predicted time untilsensor failure impacts autonomous control of vehicle 10, a time untilinclement weather is predicted to begin that may impact the operatingcharacteristic, i.e., autonomous control of vehicle 10 in this example,etc. Autonomous control system 24 may, when signaling the handoff event,also indicate to processor 12 the extent of time remaining until anestimated occurrence of the expected handoff event. In this way,processor 12 may determine the extent of time remaining until anestimated occurrence of the expected handoff event.

Processor 12 may next determine a process by which to restore awarenessof the operating context of vehicle 10 based on the determined extent oftime remaining until the handoff event. In other words, processor 12 maydetermine a process by which to inform the occupant of the operatingcontext of vehicle 10 based on the determined extent of time remaininguntil the estimated occurrence of the expected handoff event. Processor12 may also base determination of the process by which to restoreawareness of the operating context of vehicle 10 on additional factors.Processor 12 may, for example, also determine whether the primaryoccupant will be able to take control of vehicle 10 when the extent oftime remaining until the expected handoff event. As another example,processor 12 may determine a level of alertness of the occupant, and/ora condition triggering the expected handoff event. As such, processor 12may determine the process to inform the occupant of the operatingcontext based on one or more of the determined extent of time remaininguntil the expected handoff event, whether the primary occupant will beable to take control of vehicle 10 at the time of the estimatedoccurrence of the expected handoff event, a level of alertness of theoccupant, and/or a condition triggering the expected handoff event.

To facilitate determination of the occupant-based additional factors(such as whether the occupant will be able to control vehicle 10, andthe level of awareness of the occupant), vehicle 10 may include anoccupant tracking unit 26 configured to monitor the occupant. Occupanttracking unit 26 may represent a unit configured to track the primaryoccupant (and potentially other occupants of vehicle 10) to determinewhether the primary occupant will be able to control vehicle 10.

In determining whether the primary occupant will be able to controlvehicle 10 at the time of the estimated occurrence of the expectedhandoff event, occupant tracking unit 26 may determine a level ofawareness (or, in other words, an awareness level) of the primaryoccupant. The awareness level may be indicative of a level of theprimary occupant's awareness of the operating context of vehicle 10.

The level of awareness may also be referred to or indicative of a stateof the primary occupant. As such, the level of awareness may representone factor in determining an overall state of the primary occupant. Forexample, processor 12 may determine an ability of the primary occupantto take over control of the vehicle based on the level of awareness.When the primary occupant is incapacitated due to sleep deprivation (asone example) as determined through eye tracking indicative of repeatedlowering of the focus location and nodding of the head, the level ofalertness may be a controlling factor in determining that the primaryoccupant is unable to take over control of vehicle 10 at the time atwhich the handoff event is determined to occur.

In other words, processor 12 may determine a level of alertness of theprimary occupant relative to a threshold alertness level. When the levelof alertness is below the threshold alertness level by a determinedamount (e.g., 20% or more below the threshold alertness level),processor 12 may determine that the primary occupant is unable to assumecontrol of vehicle 10 within the determined extent of time remaininguntil the expected handoff event. However, when the level of alertnessis within some margin of the threshold alertness level (e.g., within 20%of the threshold alertness level), processor 12 may determine that theprimary occupant is able to resume control of vehicle 10 prior to theexpected handoff event.

The determined margin as to whether the primary occupant is able toresume control of vehicle 10 prior to the estimated occurrence of thehandoff event may vary based on the extent of time remaining until theexpected handoff event. That is, processor 12 may determine that themargin is reduced when there is less time remaining until the estimatedoccurrence of the expected handoff event, and that the margin isincreased when there is more time remaining until the estimatedoccurrence of the expected handoff event. As such, processor 12 maydetermine the process based on the interplay between the extent of timeremaining until the expected handoff event is determined to occur,whether the primary occupant is able to take over control of vehicle 10prior to the estimated occurrence of the expected handoff event, andwhether the primary occupant is determined to be sufficient alert (e.g.,the determined level of alertness is within the margin below thealertness threshold level).

To determine the level of awareness, occupant tracking unit 26 mayinclude a camera or other image capture device configured to capture oneor more images of the primary occupant. Occupant tracking unit 26 mayposition the camera (e.g., rotate the camera to a particular azimuth andelevation) to allow capture of the images of the primary occupant as theprimary occupant moves about the cabin of vehicle 10.

Using the camera, occupant tracking unit 26 may perform one or more ofeye tracking (which may also be referred to as “gaze tracking”) withrespect to the images depicting the primary occupant. More informationon eye tracking can be found in a paper by Krafka et al., entitled “EyeTracking for Everyone,” dated May 5, 2016, and another paper by Kim etal., entitled “Vision-Based Eye-Gaze Tracking for Human ComputerInterface,” dated Oct. 12-15, 1999.

Generally, eye tracking tracks the movement of the pupil through the useof corneal reflections created by projections of infrared and/or nearinfrared non-collimated light. As such, occupant tracking unit 26 mayinclude an infrared and/or a near infrared light source to create abright pupil effect similar to what is commonly known as “red eye” inimages. Occupant tracking unit 26 may generally track the primaryoccupant and then zoom in on the primary occupant's face or eyes tocapture images of at least one eye of the occupant while illuminatingthe eye with the infrared and/or near infrared light source. The extentof the corneal reflection is represented in the images of the eyes ofthe occupant in terms of brightness. The brightness of the pupil in theimage of the eyes indicates how directly the pupil is focused on theinfrared and/or near infrared light source, where higher brightnessindicates more direct focus. From this brightness of the pupil, theoccupant tracking unit 26 may determine the location at which theprimary occupant is gazing.

Although occupant tracking unit 26 is described as performing the eyetracking, occupant tracking unit 26 may perform only aspects of the eyetracking described above. For example, occupant tracking unit 26 maygenerally track the primary occupant, capturing the images of the eyesof the occupant while directing the infrared and/or near infrared lightsources in such a manner as to create the corneal reflections. Occupanttracking unit 26 may next provide the images of the eyes of the occupantto processor 12 for determining the level of awareness.

Assuming occupant tracking unit 26 determines the level of alertness,occupant tracking unit 26 may provide the determined level of awarenessto processor 12. Based on the level of awareness, processor 12 maydetermine whether the primary occupant will be able to take control ofvehicle 10 at the estimated occurrence of the handoff event.

Although described as determining an extent to which the primaryoccupant is aware of the operating context (which may also be referredto as the “driving context” in this disclosure) based on an analysis ofthe eyes and/or head of the primary occupant, processor 12 may base thedetermination on additional information, including a detected heart rateof the primary occupant, a determined tiredness of the primary occupant(through eye tracking that detects more frequent blinking or aconsistent lowering of the determined focus position, primary occupantimages that detects yawning, etc.) or any other image analysis and/orsensor signal analysis, including those that assess alcoholicconsumption by the primary occupant, etc.

Autonomous control system 24 may also, when indicating the extent oftime remaining until an estimated occurrence of the expected handoffevent, determine a condition triggering the expected handoff event. Forexample, autonomous control system 24 may determine a source orcondition in terms of whether such condition was detected visually usingcameras or via non-visual sensors, such as road sensing sensors, tiresensors, weather sensors, and the like. Autonomous control system 24 mayprovide the determined condition when specifying the extent of timeremaining until the expected handoff event, subsequent and/or prior toproviding the extent of time remaining until he handoff event. Processor12 may receive, and as such determine, that condition triggering thehandoff event.

Processor 12 may determine any combination of processes based on theforegoing factors, and perform the determined process to inform theprimary occupant (and thereby potentially increase the primaryoccupant's awareness) of the operating context of vehicle 10. Althoughdescribed in more detail with respect to FIGS. 2 and 3, processor 12 mayas one example determine a process in which vehicle 10 enters a hapticco-piloting mode when that the extent of time until the occurrence ofthe expected handoff event is within a threshold on the order of tens ofseconds, the primary occupant has been determined to be able to takecontrol of vehicle 10, the level of awareness is below a thresholdawareness level but within the awareness margin, and the conditiontriggering the event is non-visual from a sensor that senses an overlyrough road condition.

Processor 12 may interface with autonomous control system 24 to enterthe haptic co-piloting mode, requesting that the primary occupant placehis or her hands on the steering wheel. During the haptic co-pilotingmode, autonomous control system 24 may communicate via haptic feedbackin the steering wheel the roughness of the road, while also indicatingto which side the primary occupant is to manipulate the wheel tocontinue to travel to a specified destination.

In this respect, various aspect of the techniques set forth in thisdisclosure may facilitate occupant awareness in autonomous vehicles whenthe vehicle has automated some aspect of the driving. The techniques mayadaptively select a process to inform an occupant of the operatingcontext based on an extent of time remaining until the estimatedoccurrence of the expected event (and possibly a determination ofwhether the primary occupant is able to take control of vehicle 10 whenthe expected handoff event occurs, and/or the determined level ofawareness of the primary occupant). Restoration of the operating contextmay allow the primary occupant to more safely operate the vehicle uponreceiving control.

Although described as being performed by processor 12, various aspectsof the techniques described in this disclosure may be performed by GPU14 or a combination of processor 12 and GPU 14. As such, reference toprocessor 12 above may be understood to refer to one or more processors,which may include processor 12, GPU 14, a combination of processor 12and GPU 14, or any combination of various processors, some of which maynot be shown in the example of FIG. 1.

Furthermore, although described as being performed in the context ofvehicle 10 that is capable of autonomous control of the operation ofvehicle 10, the techniques described in this disclosure may apply to anytype of vehicle whether capable of autonomous control or not. That is, alevel one vehicle that cannot autonomously pilot the vehicle may performthe techniques described in this disclosure. As such, the techniquesshould not be limited to autonomous vehicles.

FIG. 2 is a block diagram illustrating various aspects of vehicle 10 ofFIG. 1 in more detail. In the example of FIG. 2, autonomous controlsystem 24 includes an operating context unit 30. Operating context unit30 may represent a unit configured to assess the operating context basedon input from on-board sensors (including cameras, tire sensors, roadcondition sensors, weather sensors, etc.) and other information, such asinformation received via local area networks (LANs), including wirelessLANs, cellular networks, inter-vehicle networks or communications, andthe like.

For example, operating context unit 30 may accept inputs from theon-board sensors and other information regarding traffic conditions(e.g., identifying an unpredictably operated vehicle), road conditions(e.g., identifying lane closures, falling debris, etc.), and weather(e.g., identifying or predicting severe weather with low visibility). Asyet other examples, operating context unit 30 may accept inputs form theon-board sensors and other information regarding vehicle 10 (e.g.,identifying communication, system, and/or sensor failures), the occupant(e.g., identifying a medical emergency concerning the primary occupant),and safety system (e.g., identifying potential evacuations, etc.).

Operating context unit 30 may determine the extent of time remaininguntil the handoff event based on the input form the on-board sensors andthe other information (where the input may generally refer to any inputfrom on-board sensors or information from other sources, such as thevarious networks or communications listed above). Operating context unit30 may provide the extent of time until the estimated occurrence of theexpected handoff event as remaining time information 31 to processor 12.Operating context unit 30 may also provide the condition triggering thehandoff event to processor 12 as condition information 33, as well as anindication that the expected handoff event is to occur.

As further shown in the example of FIG. 2, occupant tracking unit 26 mayinclude an occupant awareness unit 34 that represents a unit configuredto determine the level of awareness of the primary occupant. Occupantawareness unit 34 may interface with the various sensors (includingcameras) that track or otherwise monitor the primary occupant. Occupantawareness unit 34 may receive the eye and head tracking information froman eye and head tracking unit, heart rate information from a heart ratemonitoring sensor, respiratory rate from a respiratory rate trackingsensor, and the like and, based on this information, determine that theprimary occupant is currently sleeping.

Occupant awareness unit 34 may also utilize historical data tracking theprimary occupant to determine, as some examples, a duration that theprimary occupant has been sleeping, the current sleep cycle (e.g., deeprapid eye movement (REM) sleep, pre-REM light sleep, etc.) and otheraspects of the state of the primary occupant (including consumption ofalcoholic beverages) to determine the level of awareness. Occupantawareness unit 34 may provide the determined level of awareness asawareness level information 35 to processor 12.

As also shown in the example of FIG. 2, processor 12 may include aprocess selection unit 36 that represents a unit configured to select aprocess to inform the primary occupant of the current operating contextof vehicle 10 based on information 31, 33, and 35. Process selectionunit 36 may first determine a threshold alertness level 37 (“TAL 37”).In some examples, process selection unit 36 may be configured with astatic TAL 37 that does not adapt to the current operating context, andtherefore defines a minimum level of awareness of the primary occupantrequired to operate vehicle 10.

In other examples, process selection unit 36 may adaptively determineTAL 37 based on remaining time information 31, condition information 33and/or other information (such as the time of day, current weatherconditions, current traffic conditions, type of road on which vehicle 10is operating, etc.). For example, processor 12 may determine that a tirepressure monitoring sensor failure while vehicle 10 is traveling on arelatively remote road having little to no traffic may not require ashigh as a TAL 37 as failure of a GPS sensor on an interstate havingmoderate to high traffic levels.

Process selection unit 36 may also determine an awareness margin 39 (“AM39”). Again, in some examples, process selection unit 36 may bepreconfigured with AM 39, where such static AM 39 may represent alargest margin from TAL 37 acceptable for any extent of time remaininguntil an estimated occurrence of the handoff event. Process selectionunit 36 may be configured, for a higher static AM 39, to perform adefault process (which may include navigating to the side of the roadand stopping or stopping without navigating to the side of the road) forlonger extent of time remaining until an estimated occurrence of thehandoff event than that configured for a lower static AM 39.

In other words, a higher alertness margin may necessitate more time toensure awareness, which may result in process selection unit 36determining that the default process is to be performed when the time tothe expected handoff event is at higher levels of time (e.g., tens ofseconds or minutes). In contrast, a lower alertness margin maynecessitate more less time to ensure awareness, which may result inprocess selection unit 36 determining that the default process is to beperformed when the time to the expected handoff event is at lower levelsof time (e.g., seconds rather than tens of seconds, or tens of secondsrather than minutes).

In other examples, process selection unit 36 may adaptively determine AM39 based on remaining time information 31, determining a higher AM 39when remaining time information 31 indicates more time until theestimated occurrence of the expected handoff event, and a lower AM 39when remaining time information 31 indicates less time until theestimated occurrence of the expected handoff event. In this respect,process selection unit 36 may determine AM 39. Process selection unit 36may after determining TAL 37 and AM 39, perform the algorithm forselection of the process as outlined in more detail with respect to theexample of FIG. 3.

Although not shown in the example of FIG. 2, autonomous control system24 may also include a driver readiness unit that represents a unitconfigured to prompt the primary occupant for handoff. The driverreadiness unit may, when informed that the primary occupant is ready forhandoff, perform the handoff of one or more aspects of operation ofvehicle 10 to the primary occupant.

FIG. 3 is a diagram illustrating example operation by process selectionunit 36 of FIG. 2 in selecting a process to inform the occupant of theoperating context of vehicle 10 in accordance with various aspects ofthe techniques described in this disclosure. Process selection unit 36may first compare the awareness level (“AL,” which may refer to anotherway of the level of awareness defined by awareness information 35) toTAL 37. When AL is greater than (or, in some examples, greater than orequal to) TAL 37, process selection unit 36 may determine that theprimary occupant is able to control vehicle 10 at the estimated timethat the expected handoff event occurs, and is sufficiently alert tooperate vehicle 10 once the operating context is restored, and proceedto selection of the process based on condition information 33.

However, when the AL is not greater than (or, in some examples, is notgreater than or equal to) TAL 37, process selection unit 36 maydetermine whether the primary occupant will be able to control vehicle10 when the expected handoff event occurs by, as one example, comparingthe AL to TAL 37 minus AM 39. Process selection unit 36 may determinethat the primary occupant will be able to control vehicle 10 when theexpected handoff event occurs but not yet alert enough to currentoperate vehicle 10 when the AL is greater than TAL 37 minus AM 39 butless than TAL 37.

When the AL is less than TAL 37 but greater than TAL 37 minus AM 39,process selection unit 36 may issue one or more of an audio alert, avideo alert, and/or a haptic alert in an attempt to raise the AL of theprimary occupant such that the AL is greater than TAL 37. When the AL isless than TAL 37-AM 39, process selection unit 36 may determine that theprimary occupant is not able to control vehicle 10 at the time at whichthe expected handoff event occurs. Responsive to the determination thatthe primary occupant is not able to control vehicle 10 at the time atwhich the expected handoff event occurs, process selection unit 36 mayselect default process 41A (“def proc 41A”).

When the AL is less than TAL 37 but greater than TAL 37 minus AM 39 andin conjunction with or after issuing the audio, visual, and/or hapticalerts, process selection unit 36 may determine whether the extent oftime remaining until the estimated occurrence of the expected handoffevent as indicated by remaining time information 31 is greater than anumber of seconds (as defined by a threshold). When the extent of timeremaining is not greater than some threshold defined on an order ofseconds, process selection unit 36 may determine that there isinsufficient time to raise the AL of the primary occupant and determinethat default process 41A is to be performed.

When the extent of time remaining is greater than the threshold definedon the order of seconds, process selection unit 36 may select a class ofprocesses 41 (which may refer to all processes by which to ensureoccupant awareness) based on condition information 33. Process selectionunit 36 may classify various conditions as being better illustrated byway of video, audio, and/or haptic processes 41.

Conditions triggering the expected handoff event that may be bettersuited to video processes 41 include detecting erratic operation of avehicle traveling along the same road as vehicle 10, but not in a waythat is described or subject to analysis by a translation system ofautonomous control system 24. Another example condition that may bebetter suited to video processes 41 may include instances in whichautonomous control system 24 is unable to identify a route around anobstacle, where autonomous control system 24 may be able to generateimages and/or video for display that highlights the obstacle andpossible paths. Autonomous control system 24 may, in other words,identify conditions in which video or images of the condition triggeringthe handoff event are available, where the condition cannot beeffectively translated from sensor outputs into normal spoken language.

Conditions that are better suited for audio processes 41 include, as oneexample, when an exit from a highway is arriving shortly and autonomouscontrol system 24 has determined that the expected handoff event is tooccur at the arrival of the exit. Another example of a condition suitedfor audio processes 41 include traffic that is communicated via acommunication to autonomous control system 24 via vehicle to vehiclelinks, where the communication indicates where the traffic occurs (e.g,via GPS coordinates or mile marker) but that does not include any imagesor video of the traffic. Autonomous control system 24 may, in otherwords, identify conditions that can be effectively translated fromsensor outputs into normal spoken language and/or conditions for whichthere is insufficient video or images.

Conditions better suited for haptic processes 41 include, as oneexample, conditions concerning a roughness of the road that may bedifficult to communicate visually or verbally. Another example conditionbetter suited for haptic processes 41 include conditions concerning anamount of slippage due to snow or ice on the road, which may be clearlycommunicated through video or audio processes 41. Autonomous controlsystem 24 may, in this respect, clarify haptic conditions as those thatcannot be easily expressed through video or audio processes, but thatmay be communicated by way of haptics through the steering wheel or seatin which the primary occupant is sitting.

Based on the foregoing condition information 33, process selection unit36 may select one of processes 41A-41N (“processes 41”). Furthermore,process selection unit 36 may classify processes 41 based on thresholds43A-43D, which effectively classifies processes 41 into processessuitable for different levels of time (e.g., seconds by way of firstthreshold 43A, tens of seconds by way of second threshold 43B, minutesby way of third threshold 43C, and tens of minutes by way of fourththreshold 43D).

In the example classification shown in FIG. 3, process selection unit 36may classify processes 41A-41D as seconds-level processes 41 that may besuitable for restoring primary occupant awareness (by way of, e.g.,informing the primary occupant of the operating context) when the timeuntil handoff is on the order of seconds. Audio, video, and/or hapticalert process 41B (“A/V/H alert 41B”) may represent a process by whichautonomous control system 24 issues an audio, visual, and/or hapticalert to the primary occupant. Process selection unit 36 may selectwhich of the audio, video, and/or haptic alerts to issue based oncondition information 33. Audio query process 41C may represent aprocess by which processor 12 interfaces with user interface 22 to issuean audio query, providing information to the primary occupant. Hapticco-pilot mode process 41D represents a process by which autonomouscontrol system 24 interfaces with user interface 22 to issue hapticfeedback via the steering wheel, seat, or other surface with which theprimary occupant may interact (including tablets, smart phones or otherdevices capable of haptic feedback) that directs the primary occupant iscontrolling operating of vehicle 10.

Process selection unit 36 may select any of seconds-level processes 41when remaining time information 31 indicates an extent of time remaininguntil the expected handoff event is below (or, in some examples, belowor equal to) first threshold 43A. Process selection unit 36 may selectone or more of seconds-level processes 41 based on condition information33, tailoring selection of seconds-level processes 41 to the conditiontriggering the handoff event. In the example of FIG. 3, processselection unit 36 may classify default process 41A as conditionindependent, audio, video, and/or haptic alert process 41B as all of theaudio, video and haptic processes, audio query process 41C as an audioprocess, and haptic co-pilot mode process 41D as a haptic process.

Process selection unit 36 may classify processes 41D-41G astens-of-seconds-level processes 41 that may be suitable for restoringprimary occupant awareness (by way of, e.g., informing the primaryoccupant of the operating context) when the time until handoff is on theorder of tens of seconds. Limited augmented reality process 41E (“LTD AR41E”) may represent a process by which processor 12 and/or autonomouscontrol system 24 may interface with user interface 22 to presentlimited amounts of augment reality to present, via a heads-up display orheadset worn by the primary occupant, a marker around the conditiontriggering the expected handoff event. Video summary process 41F (“VIDSUM 41F”) may represent a process by which processor 12 and/orautonomous control system 24 interfaces with user interface 22 topresent via display 20 a video summary of the condition triggering thehandoff event and/or the operating context of vehicle 10. Narrativesummary process 41G (“NAR SUM 41G”) may represent a process by whichprocessor 12 and/or autonomous control system 24 interfaces with userinterface 22 to present an audio narrative of the condition triggeringthe expected handoff event and/or the operating context of vehicle 10.

Process selection unit 36 may select any of tens-of-seconds-levelprocesses 41 when remaining time information 31 indicates an extent oftime remaining until handoff is below (or, in some examples, below orequal to) second threshold 43B. Process selection unit 36 may select oneor more of tens-of-seconds-level processes 41 based on conditioninformation 33, tailoring selection of seconds-level processes 41 to thecondition triggering the handoff event. In the example of FIG. 3,process selection unit 36 may classify haptic co-pilot mode process 41Das a haptic process, limited AR process 41E and video summary process41F as video processes, and narrative summary process 41G as an audioprocess.

Process selection unit 36 may classify processes 41G-41K asminutes-level processes 41 that may be suitable for restoring primaryoccupant awareness (by way of, e.g., informing the primary occupant ofthe operating context) when the time until handoff is on the order ofminutes. 360 degree snapshot process 41H (“360 SNAP 41G”) may representa process by which processor 12 and/or autonomous control system 24 mayinterface with user interface 22 to present via display 20 a 360 degreevideo or image (which may also be referred to as a “snapshot”) of thecondition triggering the handoff event and/or the operating context ofvehicle 10. Video with audio narration process 41I (“VID W/AUD SUM 41I”)may represent a process by which processor 12 and/or autonomous controlsystem 24 interfaces with user interface 22 to present via display 20 avideo with real-time or near-real-time audio narration of the conditiontriggering the handoff event and/or the operating context of vehicle 10.Immersive AR process 41J (“IMM AR 41G”) may represent a process by whichprocessor 12 and/or autonomous control system 24 interfaces with userinterface 22 to present via display 20 (e.g., a heads-up display oraugmented reality headset) an immersive virtual summary of the conditiontriggering the handoff event and/or the operating context of vehicle 10.Video map trip summary process 41K (“MAP TRIP SUM 41K”) may represent aprocess by which processor 12 and/or autonomous control system 24interfaces with user interface 22 to present via display 20 a videomap-based summary of a trip that vehicle 10 is currently traveling.

Process selection unit 36 may select any of minutes-level processes 41when remaining time information 31 indicates an extent of time remaininguntil handoff is below (or, in some examples, below or equal to) thirdthreshold 43C. Process selection unit 36 may select one or more oftens-of-seconds-level processes 41 based on condition information 33,tailoring selection of minutes-level processes 41 to the conditiontriggering the expected handoff event. In the example of FIG. 3, processselection unit 36 may classify narrative summary process 41G and videowith audio narrative process 41I as an audio process, and 360 snapshotprocess 41H, video with audio narrative process 41I, immersive AR 41J,and map-based trip summary process 41K as video processes.

Process selection unit 36 may classify processes 41K-41N astens-of-minutes-level processes 41 that may be suitable for restoringprimary occupant awareness (by way of, e.g., informing the primaryoccupant of the operating context) when the time until the expectedhandoff event is on the order of tens of minutes. Audio narrativeprocess 41L (“AUD NAR 41L”) may represent a process by which processor12 and/or autonomous control system 24 may interface with user interface22 to provide an audio narration of the condition triggering the handoffevent and/or the operating context of vehicle 10. Video narrativeprocess 41M (“VID NAR 41M”) may represent a process by which processor12 and/or autonomous control system 24 interfaces with user interface 22to present via display 20 a video narration of the condition triggeringthe handoff event and/or the operating context of vehicle 10.Destination summary process 41N (“DEST SUM 41N”) may represent a processby which processor 12 and/or autonomous control system 24 interfaceswith user interface 22 to present an audio destination-based summary ofa trip that vehicle 10 is currently configured to perform.

Process selection unit 36 may select any of tens-of-minutes-levelprocesses 41 when remaining time information 31 indicates an extent oftime remaining until the expected handoff event is below (or, in someexamples, below or equal to) fourth threshold 43D. Process selectionunit 36 may select one or more of tens-of-seconds-level processes 41based on condition information 33, tailoring selection oftens-of-minutes-level processes 41 to the condition triggering theexpected handoff event. In the example of FIG. 3, process selection unit36 may classify map-based trip summary process 41K and video narrativeprocess 41M as an audio process, and audio narrative process 41L anddestination-based audio summary process 41N as audio processes.

As noted above, process selection unit 36 may classify various processes41 into two (or possibly more) time-based classes. For example, processselection unit 36 may classify haptic co-pilot mode process 41D asbelonging to seconds-level processes 41 and tens-of-seconds levelprocesses 41. Such overlapping classification suggests that each ofthresholds 43 may include both an upper bound and a lower bound (or bedefined in terms of a range) that may overlap adjacent thresholds.Processor selection unit 36 may, in this way, define thresholds 43 as arange having an upper bound and a lower bound in which various processes41 may be classified. Those of processes 41 having an expected durationthat falls within the overlapping portions of the ranges of threshold 43may be included in two or possibly more classes. Accordingly, thetechniques of this disclosure should not be limited in this respect to abinary or single classification system in which processes 41 are onlyclassified into one, and only one, class.

Moreover, within each of the time-based and condition-basedclassifications, process selection unit 36 may further distinguishbetween the sub-classes of processes 41 based on a state of theoccupant. That is, while shown as only issuing an audio, visual, and/orhaptic alert when AL is less than TAL 37, but greater than TAL 37 minusAM 39, process selection unit 36 may perform various of processes 41 inthe various above identified classes of processors 41.

The following Tables 1-3 illustrate an example of how process selectionunit 41 may select processes when AL is greater than TAL 37 (denoted as“Occupant alert and able to take control), when AL is less than TAL 37but greater than TAL 37 minus AM 39 (denoted as “Occupant not alert butwill be able to take control”), and when AL is less than TAL 37 minus AM39 (denoted as “Occupant is not able to take control”).

TABLE 1 Occupant alert and able to take control Time Remaining Occupantalert and until Handoff able to take control Within one second Defaultprocess 41A Within several seconds Audio Query 41C Audio Alert 41BHaptic Alert 41B Haptic co-piloting 41D (hands on the wheel) Limited AR(e.g. AR marker around problem area on head's up display - not shown inFIG. 3) Within in 10 s of seconds Short or sped-up video summary 41FNarrated summary 41G Short or sped-up first- person video 41F Can useopaque display (Not shown in FIG. 3) 360 snapshot displays 41H (Notshown in FIG. 3) Within several minutes Video at real-time with audionarration 41I Immersive AR 41J Within 10 s of minutes Map-based displaysummary of trip 41K Audio Narrative 41L Video Narrative 41M Destinationsummary 41N

TABLE 2 Occupant not alert but will be able to take control TimeRemaining Occupant not alert but will until Handoff be able to takecontrol Within one second Default process 41A Within several secondsAudio Alert 41B Haptic Alert 41B Within in 10 s of seconds Audio Query41C (not shown in FIG. 3) Short or sped-up video summary 41F Narratedsummary 41G Haptic co-piloting (hands on the wheel) 41D Limited AR 41E(e.g. AR marker around problem area on heads up display) Within severalminutes Short or sped-up first- person video 41I Immersive AR 41J Canuse opaque display (not shown in FIG. 3) 360 snapshot display 41H Within10 s of minutes Video at real-time with audio narration (not shown inFIG. 3) Map-based display summary of trip 41K Audio Narrative 41L VideoNarrative 41M

TABLE 3 Occupant not able to take control Time Remaining Occupant notable until Handoff to take control Within one second Default process 41AWithin several seconds Default process 41A Within in 10 s of secondsWithin several minutes Within 10 s of minutes

The foregoing Tables 1 and 2 show how process selection unit 36 mayselect various different second-level processes 41,tens-of-seconds-level processes 41, minutes-level processes 41, andtens-of-minutes-level processes 41 may be classified based on whether ALis greater than TAL 37, less than TAL 37 but greater than TAL 37 minusAM 39, and less than TAL 37 minus AM 39. Although described in thisdisclosure with respect to certain time-based classifications, thetechniques may be performed with respect to other time- orduration-based classifications that vary by orders of magnitude, orother durations.

FIG. 4 is a diagram illustrating an example of a head-mounted displaysystem 70 configured to perform various aspects of the techniquesdescribed in this disclosure. Head-mounted display (HMD) system 70includes an HMD 80 that can be worn by occupant 72, which may representone or more of primary occupant 61A or secondary occupant 61B. HMD 80may be configured to be worn by occupant 72, e.g., similar to a pair ofglasses, with a frame 84 having arms 86 extending from lens holders 88.HMD 80 is configured to communicate wireless with vehicle 10 and may actas an extension of user interface 22.

In some examples, HMD 80 may include a processor separate from processor12 along with system memory, a GPU, and other computing componentsseparate from those of vehicle 10 described above with respect to theexample of FIG. 1. In other examples, HMD 80 may simply constitute anextension of display 20, where processor 12 and/or autonomous controlsystem 24 may interface with HMD 80 to perform various of the ARprocesses noted above, such as limited AR process 41E and immersive ARprocess 41J.

FIGS. 5A and 5B are diagrams showing example forward views 90A and 90Bhaving different levels of emphasis applied in accordance with variousaspects of the techniques described in this disclosure. Forward views90A and 90B may represent either images of forward views or actualforward views viewed through HMD 80.

In the example of FIG. 5A, forward view 90A does not include anyemphasis and, therefore, represents a forward view free of augmentationby way of virtual objects. Forward view 90A shows a small vehicle (e.g.,motorcycle 92, where the bracket denoting motorcycle 92 is not displayedon the screen and is used only for reader reference) that is in the samelane as vehicle 10 and which vehicle 10 is following. In the example ofFIG. 5B, forward view 90B is the same or substantially the same asforward view 90A except that forward view 90B includes two minor virtualobjects 94A and 94B presented by HMD 80 to emphasize a condition, e.g.,the presence of vehicle 92 that is driving erratically, that triggeredthe upcoming handoff event.

FIG. 6 is a flowchart illustrating example operation of vehicle 10 ofFIG. 1 in performing various aspects of the techniques described in thisdisclosure. Initially, autonomous control system 24 of vehicle 10 mayassume autonomous control of operation of vehicle 10 (120).

After assuming autonomous control of operation of vehicle 10 (or in someinstances prior to assuming autonomous control of operation of vehicle10), processor 12 may interface with autonomous control system 24 todetermine a condition triggering a expected handoff event (which againmay refer to transferring one or more aspects of control of operation ofvehicle 10 from autonomous control system 24 to the primary occupant)(122). Processor 12 may also interface with autonomous control system 24to determine an extent of time remaining until the expected handoffevent (124). Processor 12 may next interface with occupant tracking unit26 to determine an awareness level of an occupant of vehicle 10 (126).

Processor 12 may next determine whether the occupant is alert and ableto control vehicle 10 based on the determined awareness level and thedetermined time remaining until the expected handoff event (128). Whenprocessor 12 determines that the occupant is alert and able to controlvehicle 10 (“YES” 130), processor 12 may select one of processes 41based on the determined condition and the time remaining until theexpected handoff event (132). Processor 12 may interface with userinterface 22 to perform the selected one of processes 41, and afterperforming the selected one of processes 41, handoff control to theoccupant (134, 136).

When processor 12 determines that the occupant is not alert, and as suchnot ready to assume control of vehicle 10 (“NO” 130), processor 12 maydetermine whether the occupant will be able to assume control of vehicle10 within the time remaining until the expected handoff event (138).When the occupant will be ready to assume control (“YES” 138″),processor 12 may select one of processes 41 based on the determinedcondition and the time remaining until the expected handoff event (132).Processor 12 may interface with user interface 22 to perform theselected one of processes 41, and after performing the selected one ofprocesses 41, handoff control to the occupant (134, 136).

When processor 12 determines that the occupant will not be ready toassume control of vehicle 10 in the time remaining until the expectedhandoff event (“NO” 138), processor 12 may select a default process ofprocesses 41 (140). Processor 12 may next perform the default processwithout performing handoff (142).

Again, performing handoff (or, in other words, transferring control) mayrefer to returning responsibility of control to the occupant such thatthe occupant directs operation of vehicle 10 (e.g., using a steeringwheel, pedals and other implements in traditional vehicles, or usingvarious non-traditional interfaces, such as virtual interfaces thatallow for acceleration, braking, and steering). Transferring control mayalso be only a partial transfer of control depending on thecircumstances. For example, a sensor allowing for determination of arate of travel may fail, but all other autonomous control systems forsteering and navigation may be operational. In this example, vehicle 10may transfer control of acceleration and braking but retain control ofsteering and navigation. As such, vehicle 10 may transfer control of theoperation of vehicle 10 when responsibility for one or more autonomouslyperformed operations are transferred to the occupant.

Although various aspects of the techniques are described as beingperformed by vehicle 10, various aspects of the techniques may beperformed by other devices with the results provided via a communicationprotocol to vehicle 10. That is, image or video analysis of contextualimages, gaze or eye tracking, occupant tracking, any image or videomixing (e.g., to form the composite image), and transparencydetermination for the contextual images to provide a few examples may beperformed by another vehicle, network servers or computer server farms(that is, by the “cloud”) having processors and/or GPUs in place ofthose or in addition to those of vehicle 10. As such, the techniquesshould not be limited to being directly performed by the vehicle itself.

FIG. 7 is a flowchart illustrating example operation of vehicle 10 inperforming various aspects of the techniques described in thisdisclosure. Processor 12 of vehicle 10 may first determine an expectedevent which will change an operating characteristic of vehicle 10 (150).One example of the expected event which will change the operatingcharacteristic of vehicle 10 is detecting that transfer of control fromautonomous control system 24 to the primary occupant is to occur, whichis referred to in this disclosure as an “expected handoff event.”

Processor 12 may next determine an extent of time remaining until theestimated occurrence of the expected event, and select a process of aplurality of processes by which to inform the occupant of an operatingcontext of vehicle 10 based on the determined extent of time remaininguntil an estimated occurrence of the expected event (152, 154).Processor 12 may next interface with user interface 22 and/or autonomouscontrol system 24 to perform the process by which to inform the occupantof the operating context of vehicle 10 (156).

In one or more examples, the functions described may be implemented inhardware, software, firmware, or any combination thereof. If implementedin software, the functions may be stored on or transmitted over, as oneor more instructions or code, a computer-readable medium and executed bya hardware-based processing unit. Computer-readable media may includecomputer-readable storage media, which corresponds to a tangible mediumsuch as data storage media. In this manner, computer-readable mediagenerally may correspond to tangible computer-readable storage mediawhich is non-transitory. Data storage media may be any available mediathat can be accessed by one or more computers or one or more processorsto retrieve instructions, code and/or data structures for implementationof the techniques described in this disclosure. A computer programproduct may include a computer-readable medium.

By way of example, and not limitation, such computer-readable storagemedia can comprise RAM, ROM, EEPROM, CD-ROM or other optical diskstorage, magnetic disk storage, or other magnetic storage devices, flashmemory, or any other medium that can be used to store desired programcode in the form of instructions or data structures and that can beaccessed by a computer. It should be understood that computer-readablestorage media and data storage media do not include carrier waves,signals, or other transient media, but are instead directed tonon-transient, tangible storage media. Disk and disc, as used herein,includes compact disc (CD), laser disc, optical disc, digital versatiledisc (DVD), floppy disk and Blu-ray disc, where disks usually reproducedata magnetically, while discs reproduce data optically with lasers.Combinations of the above should also be included within the scope ofcomputer-readable media.

Instructions may be executed by one or more processors, such as one ormore digital signal processors (DSPs), general purpose microprocessors,application specific integrated circuits (ASICs), field programmablelogic arrays (FPGAs), or other equivalent integrated or discrete logiccircuitry. Accordingly, the term “processor,” as used herein may referto any of the foregoing structure or any other structure suitable forimplementation of the techniques described herein. In addition, in someaspects, the functionality described herein may be provided withindedicated hardware and/or software modules configured for encoding anddecoding, or incorporated in a combined codec. Also, the techniquescould be fully implemented in one or more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide varietyof devices or apparatuses, including a wireless handset, an integratedcircuit (IC) or a set of ICs (e.g., a chip set). Various components,modules, or units are described in this disclosure to emphasizefunctional aspects of devices configured to perform the disclosedtechniques, but do not necessarily require realization by differenthardware units. Rather, as described above, various units may becombined in a codec hardware unit or provided by a collection ofinteroperative hardware units, including one or more processors asdescribed above, in conjunction with suitable software and/or firmware.

Various examples have been described. These and other examples arewithin the scope of the following claims.

What is claimed is:
 1. A method of providing information to ensureoccupant awareness in a vehicle, the method comprising: autonomouslycontrolling, by the vehicle, operation of the vehicle; determining, byone or more processors and concurrent with the vehicle autonomouslycontrolling operation of the vehicle, an expected event is to occur, theexpected event including transfer of control of the operation of thevehicle from the vehicle to an occupant; determining, by the one or moreprocessors and concurrent with the vehicle autonomously controllingoperation of the vehicle, an extent of time remaining until an estimatedoccurrence of the expected event; determining, by the one or moreprocessors and concurrent with the vehicle autonomously controllingoperation of the vehicle, a condition that triggers the expected event;selecting, by the one or more processors and concurrent with the vehicleautonomously controlling operation of the vehicle, a process of aplurality of processes by which to inform the occupant of an operatingcontext of the vehicle such that the process of the plurality ofprocesses is capable of being performed within the determined extent oftime, wherein a first subset of the plurality of processes are forhigher levels of the determined extent of time compared to a secondsubset of the plurality of processes; and performing, by the one or moreprocessors and within the determined extent of time, the process of theplurality of processes by which to inform the occupant of the operatingcontext of the vehicle.
 2. The method of claim 1, further comprisingdetermining a state of the occupant, wherein selecting the process ofthe plurality of processes comprises selecting the process from theplurality of processes based on the determined extent of time remaininguntil the estimated occurrence of the expected event, the determinedcondition, and the state of the occupant.
 3. The method of claim 2,wherein determining the state of the occupant comprises determining alevel of awareness of the occupant, and wherein selecting the process ofthe plurality of processes comprises selecting the process from theplurality of processes based on the determined extent of time remaininguntil the estimated occurrence of the expected event, the determinedcondition, and the level of awareness of the occupant.
 4. The method ofclaim 1, wherein determining the condition comprises classifying thecondition as being for communication via one or more audio processes ofthe plurality of processes, video processes of the plurality of process,and haptic processes of the plurality of processes, and whereinselecting the process of the plurality of processes comprises selectingan audio process of the audio processes capable of apprising theoccupant of the operating context within the determined extent of timeremaining until the estimated occurrence of the expected event, a videoprocess from the video processes capable of apprising the occupant ofthe operating context within the determined extent of time remaininguntil the estimated occurrence of the expected event, and a hapticprocess of the haptic process capable of apprising the occupant of theoperating context within the determined extent of time remaining untilthe estimated occurrence of the expected event.
 5. The method of claim1, wherein selecting the process of the plurality of processescomprises: selecting, based on the extent of time remaining until theestimated occurrence of the expected event and the determined condition,one of seconds-level processes capable of informing the occupant of theoperating context within seconds, one of tens-of-seconds-level processescapable of informing the occupant of the operating context within tensof seconds, one of minutes-level processes capable of informing theoccupant of the operating context within minutes, or one oftens-of-minutes-level process capable of informing the occupant of theoperating context within tens of minutes; and selecting the process ofthe plurality of processes from the selected one of the second-levelprocesses, the tens-of-seconds-level process, the minutes-levelprocesses, or the tens-of-minutes-level process.
 6. The method of claim1, further comprising: determining whether the occupant will be able totake control of the vehicle when the extent of time remaining until theestimated occurrence of the transfer of the control of the operation ofthe vehicle to the occupant occurs; and determining a level of alertnessof the occupant, and wherein selecting the process of the plurality ofprocesses comprises determining the process based on the determinedextent of time remaining until the estimated occurrence of the transferof the control of the operation of the vehicle to the occupant, thedetermined condition, the determination of whether the occupant will beable to take control of the vehicle, and the determined level ofalertness.
 7. The method of claim 1, further comprising determiningwhether the occupant will be able to take control of the vehicle whenthe extent of time remaining until the estimated occurrence of thetransfer of the control of the operation of the vehicle to the occupantoccurs, wherein selecting the process of the plurality of processescomprises determining, in response to determining that the occupant willnot be able to take control of the vehicle, a default action to beperformed autonomously by the vehicle, and wherein performing theprocess comprises autonomously performing the default action.
 8. Themethod of claim 7, wherein the default action comprises one ofnavigating to a side of a road on which the vehicle is traveling andstopping the vehicle, or stopping the vehicle.
 9. The method of claim 1,wherein the process of the plurality of processes comprises displaying adestination summary of a trip for which the vehicle is currentlyautonomously controlling operation of the vehicle.
 10. The method ofclaim 1, wherein the process of the plurality of processes comprisesissuing one or more of an audio alert, issuing a haptic alert, enteringa haptic co-piloting mode, and providing augmented reality assistance.11. The method of claim 1, wherein the process of the plurality ofprocesses comprises one or more of displaying a short or sped-up videosummary of the operating context, providing a narrated summary of theoperating context, displaying a short or sped-up first-person video ofthe operating context, utilizing an opaque display to highlight problemsor show the operating context, and displaying a 360 degree image of theoperating context.
 12. The method of claim 1, wherein the process of theplurality of processes comprises one or more of presenting real-time andnear-real-time video of the operating context.
 13. The method of claim1, wherein the process of the plurality of processes comprises one ormore of displaying a map-based summary of a trip for which the vehicleis currently autonomously controlling operation of the vehicle,presenting an audio narrative of the operating context, and presenting avideo narrative of the operating context is to be performed.
 14. Adevice configured to provide information to ensure occupant awareness ina vehicle, the device comprising: one or more processors configured to:autonomously control operation of the vehicle; determine, concurrentwith the one or more processors autonomously controlling operation ofthe vehicle, an expected event is to occur, the expected vent includingtransfer of control of the operation of the vehicle form the vehicle toan occupant; determine, concurrent with the one or more processorsautonomously controlling operation of the vehicle, an extent of timeremaining until an estimated occurrence of the expected event;determine, concurrent with the one or more processors autonomouslycontrolling operation of the vehicle, a condition that triggers theexpected event; determine, concurrent with the one or more processorsautonomously controlling operation of the vehicle, a process of aplurality of processes by which to inform the occupant of an operatingcontext of the vehicle such that the process of the plurality ofprocesses is capable of being performed within the determined extent oftime, wherein a first subset of the plurality of processes are forhigher levels of the determined extent of compared to a second subset ofthe plurality of processes; and perform, within the determined extent oftime, the process of the plurality of processes by which to inform theoccupant of the operating context of the vehicle; and a memoryconfigured to store the process.
 15. The device of claim 14, wherein theone or more processors are further configured to determine a state ofthe occupant, and wherein the one or more processors are configured toselect the process from the plurality of processes based on thedetermined extent of time remaining until the estimated occurrence ofthe expected event, the determined condition, and the state of theoccupant.
 16. The device of claim 14, wherein the one or more processorsare configured to: determine a level of awareness of the occupant, andselect the process from the plurality of processes based on thedetermined extent of time remaining until the estimated occurrence ofthe expected event, the determined condition, and the level of awarenessof the occupant.
 17. The device of claim 14, wherein the one or moreprocessors are configured to: classify the condition as being forcommunication via one or more classified audio processes of theplurality of processes, classified video processes of the plurality ofprocess, and classified haptic processes of the plurality of processes,and select an audio process of the audio process capable of apprisingthe occupant of the operating context within the determined extent oftime remaining until the estimated occurrence of the expected event, avideo process from the video processes capable of apprising the occupantof the operating context within the determined extent of time remaininguntil the estimated occurrence of the expected event, and a hapticprocess of the haptic process capable of apprising the occupant of theoperating context within the determined extent of time remaining untilthe estimated occurrence of the expected event.
 18. The device of claim14, wherein the one or more processors are configured to: select, basedon the extent of time remaining until the estimated occurrence of theexpected event and the determined condition, one of seconds-levelprocesses capable of informing the occupant of the operating contextwithin seconds, one of tens-of-seconds-level processes capable ofinforming the occupant of the operating context within tens of seconds,one of minutes-level processes capable of informing the occupant of theoperating context within minutes, or one of tens-of-minutes-levelprocess capable of informing the occupant of the operating contextwithin tens of minutes; and select the process of the plurality ofprocesses from the selected one of the second-level processes, thetens-of-seconds-level process, the minutes-level processes, or thetens-of-minutes-level process.
 19. The device of claim 14, wherein theone or more processors are further configured to: determine whether theoccupant will be able to take control of the vehicle when the extent oftime remaining until the estimated occurrence of the transfer of thecontrol of the operation of the vehicle to the occupant occurs; anddetermine a level of alertness of the occupant, and wherein the one ormore processors are configured to select the process of the plurality ofprocesses based on the determined extent of time remaining until theestimated occurrence of the transfer of the control of the operation ofthe vehicle to the occupant, the determined condition, the determinationof whether the occupant will be able to take control of the vehicle, andthe determined level of alertness.
 20. The device of claim 14, whereinthe one or more processors are further configured to determine whetherthe occupant will be able to take control of the vehicle when the extentof time remaining until the estimated occurrence of the transfer of thecontrol of the operation of the vehicle to the occupant occurs, whereinthe one or more processors are configured to: determine, in response todetermining that the occupant will not be able to take control of thevehicle, a default action to be performed autonomously by the vehicle,and autonomously perform the default action.
 21. The device of claim 20,wherein the default action comprises one of navigating to a side of aroad on which the vehicle is traveling and stopping the vehicle, orstopping the vehicle.
 22. The device of claim 14, wherein the process ofthe plurality of processes comprises one of displaying a destinationsummary of a trip for which the vehicle is currently autonomouslycontrolling operation of the vehicle, issuing an audio alert, issuing ahaptic alert, entering a haptic co-piloting mode, providing augmentedreality assistance, displaying a short or sped-up video summary of theoperating context, providing a narrated summary of the operatingcontext, displaying a short or sped-up first-person video of theoperating context, utilizing an opaque display to highlight problems orshow the operating context, displaying a 360 degree image of theoperating context, presenting real-time or near-real-time video of theoperating context, displaying a map-based summary of a trip for whichthe vehicle is currently autonomously controlling operation of thevehicle, presenting an audio narrative of the operating context, orpresenting a video narrative of the operating context is to beperformed.
 23. A device configured to provide information to ensureoccupant awareness in a vehicle, the device comprising: means forautonomously controlling operation of the vehicle; means fordetermining, concurrent with autonomous control of the operation of thevehicle, an expected event is to occur, the expected event includingtransfer of control of the operation of the vehicle from the vehicle toan occupant; means for determining, concurrent with autonomous controlof the operation of the vehicle, an extent of time remaining until anestimated occurrence of the expected event; means for determining,concurrent with the vehicle autonomously controlling operation of thevehicle, a condition that triggers the expected event including transferof the control of the operation of the vehicle to the occupant; meansfor determining, concurrent with autonomous control of the operation ofthe vehicle, a process of a plurality of processes by which to informthe occupant of an operating context of the vehicle such that theprocess of the plurality of processes is capable of being performedwithin the determined extent of time, wherein a first subset of theplurality of processes are for higher levels of the determined extent ofcompared to a second subset of the plurality of processes; and means forperforming, within the determined extent of time, the process of theplurality of processes by which to inform the occupant of the operatingcontext of the vehicle.
 24. A non-transitory computer-readable storagemedium having stored thereon instructions that, when executed, cause oneor more processors to: autonomously control operation of a vehicle;determine, concurrent with the vehicle autonomously controllingoperation of the vehicle, an expected event is to occur, the expectedevent including transfer of control of the operation of the vehicle fromthe vehicle to an occupant; determine, concurrent with the vehicleautonomously controlling operation of the vehicle, a condition thattriggers the expected event including transfer of the control of theoperation of the vehicle to the occupant; determine, concurrent with thevehicle autonomously controlling operation of the vehicle, a conditionthat triggers the expected event including transfer of the control ofthe operation of the vehicle to the occupant; determine, concurrent withthe vehicle autonomously controlling operation of the vehicle, a processof a plurality of processes by which to inform the occupant of anoperating context of the vehicle such that the process of the pluralityof processes is capable of being performed within the determined extentof time, wherein a first subset of the plurality of processes are forhigher levels of the determined extent of compared to a second subset ofthe plurality of processes; and perform, within the extent of timeremaining until the estimated occurrence of the expected event, theprocess of the plurality of processes by which to inform the occupant ofthe operating context of the vehicle.